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ABSTRACT 

The political processes involved in an academic 
program evaluation are discussed. The focus is on how evaluative 
information is politically linked to programmatic decisions within 
the context of various organizational and evaluative utii-ization 
models. To examine the explanatory power of the various theories and 
models, information from a program evaluation conducted at a small 
public liberal arts university is examined. The scope of this program 
evaluation is as follows: to obtain a systematic view of the relativ-e 
strengths and weaknesses of each department, and to use the feedback 
to improve* the programs; to have participants rank order each of the 
19 academic programs on nine criteria; and to ^airget future resource 
allocations to program changes. Attention is directed to the 
following issues that may affect how evaluative information is linked 
to programmatic decisions: stages of the evaluation process, a 
distinction between the reasons and motives for the evaluation, 
involvement of the participants/decision-makers in the evaluation 
process, communication of the results, organizational setting, 
recognition of the subjectivity 



of the evaluation, and the role of 



the institutional researcher. A list of evaluation criteria, an 
evaluation form, and a bibliography are appended. (Author/SW) 
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" ABSTRACT 



This paper discusses the political processes involved in o 
an academic program evaluation. Specifically, *the paper 
focuses on how evaluative information is p^olitically 
linked to programmatic decisions within the context of 
variovis organizational and evaluative utilization models. 
Data exoeriences from a program evaluation conducted at 
a small' (N=2bOO students), public, liberal arts university 
in the Southeast is use^d in examining the explanatory power 
of the various theories and models. 
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It has been stated recently that academic and administrative 
program evaluation is the latest growth indu^t^ry in higher 
education (Wilson, 1982). The reasons for- this "boom" are 
numerous (Harcleroad, 1980). Foremost among the current reasons 
given for academic program evaluations are: 1) the need for 
colleges and universities to better respond to declining 
enrollments and restricted budgets; and 2) demands for . 
accountability from external agencies. Interspersed with the 
above reasons is the idea that knowledge about a program's 
strengths and weaknesses will lead to an improved program. 

In conducting an academic program evaluation, an institution 
may select from a number of evaluation models (Anderson & Ball, 
1978; Craven, 1980; Dressel, 1976; Grant, 1978; House, 1980; 
Wilson, 1982). Typically, the models employ a self-study or 
"blue-ribbon" committee structure utilizing a variety of 
^assessment techniques (e.g., case studies, ratings, observations, 
V-nterviews, record analyses). 

Qne concept that is receiving increasing attention, and which 
was used for this study, is the academic audit. The idea for 
applying the business concept of an audit to certain situations in 
higher education is not new. Harcleroad and Fichey (1975) have 
advocated the application of processes inherent in the business 
audit to the study of institutions of higher education. As 
explained by Wirsig (1978) the audit is a regular, systematic 
assessment of the university's programs and practices and their 
impact on "society" and the "envir9nment" . The relevant 
dimensions for the concept are that the assessment is regular and, 
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systematic and that it investigates the program in relation to the 
total university and its mission. ^ 

Whether it is an academic audit or other evaluation model, 
there are certain key elements of the assessment process that are 
necessary if an evaluation is t^be effective (i.e., utilized). 
Central to these elements is the understanding of the political 
processes involved in an academic program evaluation. The purpose 
.. of this paper will be to focus on how evaluative information is 
linked politicaly to programmat'ic decisions within the context of 
an academic program evaluation conducted at a small tn=2500 . 
Students), public, liberal arts university in the southeast. 



Context 



The academic program evaluation procedure employed in this 
paper was adapted from a model developed by Shirley & Caruthers 
(1979). In that model a number of evaluative criteria, ranging 
from central ity of program to service of the program in the 
community, were used to assess the perceived strengths and 
weaknesses of academic programs in the university. The procedure 
involved participation by every academic department and required 
that each program be assessed on specified criteria. Since the 
evaluation of academic programs is a much more involved process 
than just simply placing the program on a particular grid or 
developing a rating for that program, it was considered to be an 
important part of the methodology . of this study to actively 
involve all participants, in this case all of the department 
t chairpersons. 
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The first step in the evaluation process was to insure 
broad-based participation and, hopefully, wide-range support for 
the project. It was felt to be highly important from a polit?cal 
standpoint to have all chairpersons participate, since the 
judgments and outcomes could potentially have very significant 
impacts- on their departments. If some chairpersons did not 
participate, their likelihood of supporting the process au . being 
wilWng t adhere to the recommendations "^would be substantially 
reduced . 

In order to minimize the amount of time engaged in the 

project and to indicate to the chairpersons that their time was 

perceived as valuable/ they were not asked to create an 

instrument. Rather r the chairpersons were presented with a model 

■ 

for the evaluation of academic programs ^ adapted from Shirley and 
Caruthers (1979), and asked to review and modify it. 

The project was initiated and coordinated by the Office of 
Academic Affairs. The responsibilfity for presenting the proposal 
to all department chairpersons and soliciting their participation 
was placed with the Assistant Vice Chancellor for Academic 
Affairs. 

prior to meeting with the chairpersons, the Assistant Vice 

J, 

Chancellor solicited the support of the University's Planning 
Council. As the group responsible for determining long-range 
goals of the institution as well as short-range strategies and 
tactics, it was important for the Council to actively 'encourage 
the evaluation of academic programs, it was felt that an 
assessment of the relative strengths and weaknesses of the 
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academic programs would be invaluable in determining how best to 
^utilise resources in order to achieve insitutional and 
departmental goals. The Pla-nning Council agreed to request that 
' an academic program evaluation of all departments be conducted. 

After the Planning Council approved the request for the 
evaluation, meetings were called by the Assistant Vice Chancellor 
to ask the chairpersons to consider participating in such a ^ 
project. Considerable lobbying and behind the scenes negotiations 
were conducted in *order to get the approval of key chairpersons. 
To gain concensus from all participants meant that they had to 
understand and agree on the importance of the evaluation. 

The most significant hurdle was the perceived versus actual 
motives behind engaging in the project. Several chairpersons were 
convinced that the "real" motive in initiating the evaluation 
project was to target several weak departments for elimination. 
In recent years departments with very low numbers of graduates had 
been c»losely scrutinized by state level agencies. Consequently, 
the Academic Affairs office was seen by some as having a hidden 
agenda, in the evaluation project. Assurances from the Office of 
Academic Affairs helped to dispel that concern. By engaging in 
lengthy interaction between chairpersons and the Assistant Vice 
Chancellor, an understanding was reached as to the reasons for the 
project and the likely outcomes. 

The primary^ reason for the project was to obtain a 'systeroatic 
view of the relative strengths and weaknesses of each depu<rtment. 
Using the information for resource, allocation was seen as viable 
purpose.. Each department was expected to use the feedback gained 
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from the project' to improve certain aspects of their program. It 
should be noted that this particular university has /witnessed 
enrollment growths ranging from 10-14 percent during the last • 
thvee years. Consequently, there has been little need to raise 
the spector of dismissal or termination of programs for financial 
exigency. The thrust of the project was to identify which 
departments were stronger and which departments were weaker in 

order to be given further suppoi^t and encouragement to grow. 

' ■ « . ■ " 

Methodology 

Once the department chairpersons agreed to the reasons for 
the evaluation and the evaluation model/ the discussion turned to 
the proposed methodology. Chairpersons were invited to make 
contributions to and modifications of the evaluation instrument . 
being proposed. The only limitation or restriction placed on the 
modification was that all programs were to be evaluated; A number 
of suggestions were introduced by the chairpersons that- 
substantially modified the types of criteria that were considered 
appropriate for evaluation. These criteria were tailored to the 
characteristics of the institution under study. Once a particular 
criterion (e.g., centrality to the mission of the institution) 
was selected, all of the chairpersons dijncussed and reached some 
consensus as to the meaning of that particular criterion. 
Specific qualitative and quantitative data were identified that 
would assist i^ evaluating the respective departments on the 
criteria. In each case, the data sources were agreed upon by 
chairpersons as being reflective of a particular selected 
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criterion. This process of concensus wa^ essential to ensuring 
the full support of the chairpersons. The Office of Institutional 
Research was instrumental- in helping to identify data s^ources and 
in compiling the relevent information for the chairpersons. 

Having adopted the instrument, the participahts were asked to 
rank order each of the 19 academic' programs on the nine identified 
critei-ia. All nineteen chairperons submitted their completed • 
rating sheets to the Office of Institutional Research for 
processing and analyses. (See Appendix for sample evaluation • 
form.) / 



-/■ 



Data Analysis 

The data analyses employed for this study wer'e simple 
descriptive statistics. Since the ratinas were essentially 
ordinal comparisons, a median score was determined for each 
department on each criterion. The median scores were then rank 
ordered to determine the relative position of each department on 
each criterion. The rank ordering for each of the criteria for 
the 19 departments was then summarized into quintiles. The 
results thus identified those academic programs considered to be 
\ in the top and bottom 20% of the departments on each critierion. 

The results were presented to the department chairpersons 
some two months after they had completed the project. The 
information was initially given to the chairpersons in advance of 
a scheduled department chairperson's meeting to afford each member 
the opportunity -to review the results and be prepared to discuss 
them. Because some four months were spent in developing the 
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support and wiUin^ess participate in the project by the v 
department chairpersons, once the results were published and 
distibufed to the- chairpersons, very f.e* individuals questioned or 
in any way "^failed to accept the results of the evaluation process. 



Results ' .... , * * . 



Since the critical aspect of this evaluation process related 
• . ■ - ' ■ ' . ♦ ' i - 

tp the intent or expected uses that the results would be put to,..; 

considerable time was spent discussing what the results -meant for 
each department. Chairpersons were encouraged to- make any progrc^ni 
improvements suggested "by the , results . In part, future resource^ ^ 
allocations were earmarked, for such changes. All of the 
chairpersons appeared to accept the results of the study -as ^ 
" ' . meaningful measure of the relative position of their, department in 
the university. (See Table l/ That is, each' individual department 
chairperson was able to obtain a reading of the perceived 
strengths *and weaknesses of their department, as seen by their 
colleagues. The feedback .to the individual departments as to 
their relative strengths and weaknesses has clearly justified and 

o ■ • . 

indicated the vs»lue of engaging in such a project. ^ 

Since the\rJject was initially requested by thfe" University 
Planning Council, they received the report from the {Participants 
in the project and incorporated the results into goal statements 
for the institution. In particular, one of thfe goals of the 
> . institution- was to develop quality undergraduate programs. A 

' special emphasis has been placed in this year's budgeting process 
on those programs that were perceived below par compared to other 
departments. ^ / 



Also, the results were scrutinized jjf the Planning Co.uncil to 

determine why certain departments were repeatedly rated as being 

very low. It was found on review that most of the departments 

that, were down graded by the chairpersons were departments that 

had recently been established at the university. These new 

•programs were seen by some department chairpersons as being 

.competitive for already scarce respurces. To a certain' extent , 

chairpersons were looking out for their own self-intereSts . 

From an administrative standpoint the results of the study 

have afford*ed the Office of Academic Affairs fnd the Executive 

bff.icVr of the University the opportunity to identify the relative 

Btrft/igths and weaknesses of the academic programs on campus. 

StrWgies can Now be developed by the administrative staff that 

would insure' that weaker programs would be brought up? to a more . 

.satis.factory level, while maintaining and encouraging the growth 

and development of the sttongef programs. The evaluation also 

afforded membe'rs of' the administrative staff^with information as 

to which- areas' departments were relatively strong, thereby 
' ^ " • * ' ' ■■ 

• allowing the publication and en^loitation of those strengths in 

v'ariqus media outidts to the benefit of the university. 
" " ' The 'u^^u mess of the project has far outstripped the 
expected benefits in engaging in such .an evaluation procesS. The 
'^ap^ector qE possible layoff* and filings as a result of this 
■"-evaluation process did not have to be confronted.* However, if the. 
'institution were to suddenly experience a very significant down 
^urn in enrollment or in f inancial support , the groundwork has 
been laid as to identify the relative, strengths and weaknesses of 
the academic dej^artiFnents, 
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DISCUSSION 



In analyzing how evaluative information is linked"^ to 
programmatic decisions at this particular institution, seven 
issues were identified as extremelyVmportant . These issues 
included: 1) stages of the evaluation process; 2) distinction 
between the reasons and motives for the evaluation; 3) involvement 
of the participants/decision makers in the evaluation process; 4) 
communic^on of the results; 5) organizational setting; b) 
j recognition of the subjectivity of evaluation; and 7) role of the 
institutional researcher. 

The above seven issues are by no means the only major issues 
in the utilization of evaluative information. Braskamp and Brown 
(1980) have suggested that major issues in the utilization of 
evaluative information include a broad definition of utilization, 
a concern for utilization beginning with the initial meetings, an 
active role of . the evaluators, the relevance of the evaluative 
information, the relationship betW3en the evaluator and intended 
audience, and the importance of the communication -process . In 
addition to Braskamp and Brown (1980), the reader is referred to 
Anderson and Ball (1978), Craven (1980), and Wilson (1982) for a 
complete discussion o^ the issues involved in the utilization of 
academic program evaluation information. • 



« 



(9) 



ERIC ^ / ; 13 



stages of the Evaluation Process 

In the case study presented a\f^e, the evaluation process was 
a "poltical- endeavor. As such, conflicting attitudes on the part 
of certain participants with regard to the evaluation had to be ^ 
dealt with in such a manner as to eventually arrive at some sort 
of consensus. More important, however, was the fact that 
consensus had to be reached at different stages in the evaluation. 
Failure to reach consensus at critical times would have 
jeopardized the evaluation. In the experience of the above 
institution those critical times were: 1) the endorsement by the 
chairpersons that the evaluation was necessary and that it was 
important to participate in. the project; 2) the approval by the 
chairpersons as to the methodology; 3) the communication of the 
results to the chairpersons and Planning Council; and 4) the 
decisons made by the chairpersons and Planning Council in response 

to the evaluation. 

By far, the most crucial dimension of this activity as it 
relates to evaluation was the concensus phase. As Weiss (1972 
p. 318) points out: "the basic, rationale for evaluation is that it 
provides information for act ion .. .unless it gains serious hearing 
when program decisions are made, it fails its major purpose". If 
the evaluation is able to get its "serious hearing," the extent of 
political activities in the latter stages may be decreased as was 
the case presented above. The results of the evaluation served 
to reinforce pre-existing opinions about the various programs. 
Thus, the consensus was fairly high concerning decisons about the 

various academic programs. 

O ' ■ (10) 
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Distinction Between the Reasons and Motives for the Evaluation 

The chairpersons questioned the motives of the administration 
and subsequently the reasons, for the evaluation. Thus, the first 
political activity of those responsible for the evaluation (i.e., 
the Office of the Vice Chancellor for Academic Affairs) was to 
clarify these reasons to the chairpersons. 

There appear to be three general reasons why program reviews 
are initiated. The first is that reviews can provide departments 
with information for program plannning and improvement (Clark/, 
1977). Evaluations of this kind are often met with wide 
acceptance, due to their built-in appeal. Faculty, students, 
deans, and chairpersons are provided with the opportunity to 
express their satisfaction or dissatif stion with various aspects 
of the department. The participants are eager to believe that if 
"areas of concern" are identified, attempts will be made to 
correct the situation. The evaluaton can act as a catalyst to 
spur necessary changes to insure high quality programs. 

One aspect of the evaluation was to incorporate Hollowood's 
(1979) life-cycle curve to identify the relative stages and 
positions of the departments (See Figures 1 and 2). Departments 
could be identified into one of four stages—new, growth, stable, 
declining~and one of five pos i ti t ions-weak , tenable, favorable, 
strong, or dominant. Identifying the departments according to 
this matrix proved to be very effective in describing a particular 
department • 
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The second reason for program reviews is to provide 
university administrators with information for budgetary and 
planning purposes (Clark, 1977). Universities are recognizing that 
state and private support will not continue at previous levels, 
and that resource reallocation may be the only way to initiate ne 
programs or changes. Program reviews have also been used for 
program retrenchment or abandonment (Benecerraf et al,, 1972; 
Shirley and Volkwein, 1978). Some observers (Gilmour and Beik, 
1977) have predicted that such efforts will continue. However, 
the use of program reviews for program abandonment has been 
questioned Smith (1979). 

The third general reason for program reviews is for use by 
external groups (Clark, 1977). Such groups include accreditation 
agencies and state governing boards. 

In addition to the above reasons there may be several 
underlying motives why program reviews are conducted. One such 
motive could be a reaction to increased criticisms of the 
department. The criticisms levied against departments include: 
their decentralization of authority, creation of political and 
social blocks within the university, and general erosion of the 
university's effectiveness (Anderson, 1968; Benezet, 1977; Dressel 
et al., 1970; Dressel and Reichard, 1970). Additional motives may 
be a desire by the administration to "maintain" or "acquire" more 
control over the department, to create an illusion that the 
university is responding to requests for accountability, and a 
desire to "shake-up" selected departments. 
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It was found to be the case that the motives for the 
evaluation had to be separated from the reasons for the 
evaluation. This entailed much political discussion and 
reassurance from the administration to the chairpersons before 
they would endorse the evaluation project. 

Involvement of the Participants/Decison Maker s in the Evaluation 
Process 

One of the major themes that has been continually repeated in 
the literature on the utilization of evaluative information 
(Fields, 1976; Braskamp and Brown, 4980) is the importance of 
involving the participants affected by the evaluation in the m 
evaluation process itself. As pointed out by Richardson (1980 p. 
104-105): "Individuals and groups are more likely to involve 
themselves in planning and budgeting deliberations if they 
perceive that they or their primary organizational subunit will be 
directly affected, either favorably or adversely, by the resultant 
decisions." 

In the case study presented above two important issues 
related to the involvement of the participants were identified. 
First, participation in the evaluation process was intitially met 
with some resistance. In the beginning the reasons for 
nonparticipation in the evaluation was predicated on philosophical 
opposition to program evaluations. Spme persuasion was needed to 
convince certain participants of the merits of the evaluation 
process. Secondary reasons for nonparticipaton were feelings that 
the evaluation was a waste of time and ^.hat the results would not 
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be utilized. Although these* concerns are not uncommonr they do 

attest to the fact that faculty participation in plannning and 

evaluation activities are inversely related to the organizational 

level at which these activities are conducted (Richardsonr 1980). 

The second issue of involving participants in the evaluation 

was the realization of the importance of t^e communication 

networks.. As indicated by Smock (1982^ p. 70): " Whether or not 
» 

the judgements lead directly to decisions or are linked to the 
decision-making process in a more indirect and less observable 
manner depends more upon the network of \ people designed to 
implement the systftm of program evaluation than on the results 
themselves." 

Involving the participants in the evaluation process requires 
a good understanding of the evaluation audiences and the organized 
structure of the network (Smock, 1982). Richardson (1980 p. 105) 
has pointed out that organizations possess both formal and 
informal communication networks, and that "an effective 
communication plan recognizes this fact and tries to utilize both 
types of networks in transmitting information to organizational 
participants". 

In the experience of the institution under study, it cannot 
be over emphasized the importance of involving key audiences in 
the evaluation process. Many political problems were resolved by 
including the participants in the evaluation process from the 
beginning and by utilizing, in particular, the informal 
communication networks. It is our belief that failure to involve 
the audiences and using the informal networks would have seriously 
compromised the project. > 
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Communication of the Results ^ 

' The fourth major element in linking the evaluative 
information to programmatic decisions was the communication of t 
results. Paralleling the importance of involving the 
participants in the evaluation process v/as the means for the \^ ' 
communication of the results. Here emphasis was placed on 
understanding the political processes involved between the sender 
and receiver of the evaluation results as well as how the 
information was presented. . 

As indicated in the preceeding section, organizations 
' typically have informal communication networks. These network^ 
are o£ten more important in the dissemenation of information than 
the formal structures, in that they provide for open-ended, 
two-way communication in which the evaluation information and its 
implications can be explored rather than presented (Datta, 1981). 
In the case study, this informal two-way communication allowed fo- 
the continual presentation of data, and exchange of information, 
thereby reducing the possibility of confrontation when the formal 
results were presented. The importance of this two-way 
communication was highlighted by Braskamp (1982 P-58) in his 
discussion of how an evaluation system cannot exist without an 
effective information system: 

The communication network among the audiences 
should be the core of an evaluation system. 
Evaluation is undertaken in a. social and political 
environment in which various groups have vested 
interests in the evaluation process. . . If an 
evaluation is be used by these groups in their 
deliberations, discussions', and policy making, the 
evaluaton system must be designed to m imize the 
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communication among these audiences. In designing 
such a system ,two guidelines regarding Q#mmuncation 
must be used. First, audiences and their 
information needs must be identified. However, 
pinpointing specific pieces of information to be 
used for specific decisions is not the goal. 
Rather, the intent is to establish an envifonment 
in which audiences become involved in as much of 
the evaluation process as possible, including 
making decisions about the criteria to be used in 
the evaluation, the data sources to be used, and 
interpretation and transmission of the evaluativc 
inf ormatioi?. Involvement may promote interest in - 
the evaluation and increase the ust of the 
information . ' . 

Numerous researchers (Anderson and Ball, 1981; Braskamp and 
Brown, 1980; Breckell and Aslanian, 1974; Havelock and oth- - 
1973) have investigated various factors which influence the 
communication and utilization of results. Amc^ng the key factors 
which were identified in the case study were: 1) sharing of 
roughdrafts and preliminary thoughts with key participants before = 
making the final presentation; 2) presentation of periodic 
informal reports; 3) asking the participants to assist in the 
interpretation of the findings; and 4) the credibility of those 
persons responsible for processing and making the formal 
presentations of the'results. 

Organizational Setting * /- 

Various organizational models have been presented (Backarach 
and Lawler, 1980; Baldridge, 1971; Robbins, 1976; Thompson, 1967) 
to describe the decision making processes in organizational 
settings. The three mo^t common types of models applicable to the 
university setting are the bureaucratic, rational, and poltical 
models (Baldridge, 1971). Of the three, the p' 'tical model 
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appears to have the most explanatory power in describing r 
underltanding f and predicting behavior in higher education (Tonn^ 
1980) • Ton'n (1980 p, 123) contends that the poltical model is 
better because the assumptions that must be made to use the 
alternative models are of questionable validity in their 
4 application to complex organizational settings: In complex 
organizations r where the perfect knowledge required by the 
rational models does not exitr judgment and compromise frequently 
supersede computational and bureaucratic forms of rational 
decision making. Decisions are not made solely by impartially 
determined formulae or by straightforward appeals to hierarchy r 
but are subjectively negotiated." ^ 

The use of politics in higher education is often viewed with 
disdain. Higher education personnel sometimes like to think that 
they or their organization are above having to resort to 
••politics. Often the word "politics" conotes •'trickery, playing 
games, or deception". A more accurate definition of polit^al 
behavior, however, would be: "any behavior by organization, members 
that is self-serving (Robbins, 1976, p. 64)." Totfn (1980) points 
out that this itself is not power, but requires power to attain 
it^ end. The- question thus becomes not "who has power", but ri 
"which ends are to be achieved." The powerful person is able to 
be influential in negotiating decisions. In this definition 
ev.ery department chairperson and dean would exhibit political 
behavior to obtain appropriate resources for his/her organization. 
It would be in their best self -interests . « 
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The exerting of political behavior^by participants in an 
academic program^evaluat ion should consequently corae as no 
surprise (Guskinr 1980 ) • As Braskamp and Brown (1980, p. 93) 

pointed out: 

•Evaluation is often so intertwined with 
organizational politics that evaluation itself 
might be labeled a political activity* This means 
that ttie relationship between the evaluator and key 
program staff r and the evaluator 's understanding of 
the organization in its internal and external 
^ political environment r are critical Cor successfv 
utilization. It is rare that 'an effective 
evaluator conducts a successful evaluajtion after 
having had only one meeting with key personnel. An 
effectively utilized evaluation is more likely to 
result from a process that includes a highly 
interactive consulting relationship with the key 
decision • makers • 

This was found to be the case in the^ evaluation process at the 

university described above. 

After overcoming the intitial objections to the academic 

program evaluations , most of the chairpersons in the-'study came t 

view the evaluation process as fair and began to place some trust 

in the. evaluators. One reason for this trust may be due to the 

fact that the organisational setting and structure was taken into 

account. As Smock (1982, p* 70) pointed out: 

Evaluations can be powerful inducements for 
change or Relatively ineffectualr depending on the 
organization of the network through which the 
evaluation information will* be collected, 
considered, and acted upon* Not only is it a 
matter of how the network is structured, but also 
who is selected to be part of it* Democratically 
elected committees will behave differently than 
appointed or voluntary committees* Recommendations 
from a ^?ody composed of academic heavywe ights will 
be different than those coming from more broadly 
respresentative bodies. While reliability and 
validity are yardsticks of quality that can be 
applied to data, trust and fairness ^ e similar 
yardsticks by which to judge the stt ure. , 
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^Program evaluation systems that are sensitive to 
the power arrangements on campus, aware that there 
can. seldom be top much communication and designed 
so that sound judgments are forthcoming are likely 
to measure high on the trust and fairii.3ss 
yardsticks. After all, judgment is at the heart of 
evaluation, and there is a question of, credibility 
in all judgements. 

■ 

Recognitition of the Subjectivity of Evaluation 

The question of credibility in judgments is perhaj?s a 
question of the subjectivity of evaluation. Evaluation by its 
very name implies the making of judgments. These judgments can be 
"objective", "subj.ective" , or a combination of the two. The 
distinction between the two types of judgments is typically based 
on the nature of how the judgments ,are formulated. Objective 
judgments are normally based upon quantitative data, while 
subjective judgments are typically based upon qualitative data 
(Astin and Solmon, 1979; Guba, 1978). In practice, any judgement 
must ultimately be subjective (Guttentag, 1973). 

Every institutional researcher has witnessed examples of the 
variety of inferences that can be drawn from a seemingly 
"objective" set of data. The issue of evaluation should not be one 
of eliminiating subjectivity, for that is impossible, but rather 
how to control and structure the degree of subjectivity. Dressel 
^(1982) argued that the subjectivity/objectivity in evaluations is 
intricately tied to values inherent in colleges, universities, 
departments, and decision-makers. He contends (p. 38) that: "Even 
when there is a conscious recognition of specific value 
commitments, this recognition seems only to intifoduce a subjective 
element into evaluation, which decreases ^he acceptability of the 
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results to others. Objectivity in evaluation is- a value^in. 
achieving valid results and in gaining acceptance of them." Under 
Dressel's (1982) concept of evaluation, objectivity i3 a desirable' 
attribute of an evaluation report, however, ^it is difficult to 

.achieve if any attempt iS made to interpret the data and explore ^ 

^its implications. t> ? 

Kuh (1981) has argued that in the measurement of quality, a 
holistic approach should be adopted that includes both ^ ^ 

qualitiative and quantitative approaches. Excluding either 
approach would tend to overlook important elements and components 
of the academic program. Thus, in the program evaluation under 
discussion, chairpersons had to be assured that a variety gf 
information would be used in the analysis of their i^rogram and 
tliat the subjectivity of the ratings would be considered in the 
context from which they were given. 

Role of the Institutional Researcher 

The establishment of the analytical and information 
requirements for conducting a systematic academic program 
evaluation is an appropriate role for the institutional 
researcher. The specification of the appropriate data collection 
instruments and identification of appropriate statistical 
techniques used to determine and measure the quality, ^parameters of 
the academic .programs is a role easily fulfilled by the 
institutional .researcher. 

The institutional researcher is usually a person with broad 
knowledge of the attendant considerations necessary for an 
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ef.fe.ctive, audit. First, the institutional researcher typically 
has no vested' intere-st in the outcome of the evaluaton. Second, 
the IR director is usually the person with experience in 
conducting evaluations within am investigatory framework outside 
of the political "^dynamics of the academic program. Third, tbe IR 
office can better coordinate *data collection and analysis witJ- 
the framework of a professional institutional research activity 
without it dettacting from routine activities. Faculty for the 
most part require released time to devote to a project of this 
nature, whereas the IR office couLd consider this project as 
routine. Fourth, the- IR person can contribute to the credibility 
and acce-ptance of the findings due to his/her relative detachment 
in the evaluation process. Fifth, the IR person has. 5 more , global 
concept of the overall administrative aspects of academic programs 
than the average faculty person. Sixth, the institutional 
researcher is in a unique pi^sition to'fulfill the role as a 
teacher--taking an active role in getting their data understood 
and utilized (Wise 1980). 

In summarizing the role of the institutional resea-cher in 
program reviews af three major insitutions, Munitz and Wright 
(1980) identified the following five general responsibilities that 
the institutional researcher played in establishing and supporting 
the academic program evaluation processes at their respective ^ 

institutions: 

1. Institutinal researchers participated with key 
administrators and faculty committees in the 
initial design of the program evaluation 
processes and supporting materials. 
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2. 



Staff members wepe respofisible^or overseejng 
(if not acftually doing) the--c^solidaton of 
planning data from various admnistrative data 
bases ,and for ^distributing the daCa schedules 
and accompanying f orias 'and instructions to the 
departments and colleges. 



3, 



4. 



The institutional- research of frices usually 
collected the departinental and. college evaluaton 
responses/ plans, and budget , requests ^ They, 
theji analyzed and sum irized the material for 
use by administrative and .faculty leaders in 
arriving at planning decisions . 

Tt\e institutional research staff .also 
participated to some degree in the design of 
» budgetary measures and,^ through analyses b'ai 

on these measures , in recommendations of 
initial allocations consistent with ov^erall 
budget priorities. - a , 

5. Instit^utional researchers worked closely with 
I 'administrative and faculty groups in ' 
recommending and implementing further ^ 
refinements in evaluation processes, statistical 
schedules, and instructions. 

To some degree or another the Institutional Research office in the 
institution under study was involved in each ^ the above 
respbfisibilities . . ' - ; 
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APPENDIX A 
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